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Introduction 

A mentoring program in Detroit, Michigan provides services to 300 low- 
income students each year. A tutoring program in Olympia, Washington 
serves over 1,200 students across the district. And a mathematics inter- 
vention in El Paso, Texas, helps 230 students complete Algebra 1 by the 
end of the eighth grade. Each of these programs offers a public good by 
helping students overcome barriers to educational progress. These pro- 
grams typify thousands of similar but different efforts across the country, 
some supported by public funds, others privately. While the prevailing 
notion is that these programs play an important role in the preparation of 
at-risk and other underrepresented students for college, there is little em- 
pirical evidence related to the actual success of these programs (Gandara 
and Biel, 1999; Swail and Perna, 2002). 



Rarely do stakeholders ask about the effectiveness of outreach programs 
or whether they are an efficient use of tax dollars and philanthropic 
funds. But as government budgets continue to be constrained and philan- 
thropic investment gets more competitive, there is a growing acknowl- 
edgement of the need to look at the cost/benefit of these programs and 
whether the investment is worth the outcomes. 

This paper is excerpted and updated from a similar chapter written by 
EPl President Watson Scott Swail for a recently released SUNY Press 
book. Preparing for College, edited by William Tierney, Zoe Corwin, and 
Julia Colyar of the University of Southern California. Value Added consid- 
ers issues related to the complex proposition that the cost of program de- 
livery is directly and positively tied to the ability of programs to success- 
fully enable students to get into college. Background information on the 
funding of these programs and the competition for limited fiscal re- 
sources are included, and readers are introduced to cost analysis as a 
method of answering these questions. 




Partially because cost analysis can be an extremely complex analytical 
process, it is seldom used in the education arena. However, when used 
properly it can be a very powerful tool in defining the true social and eco- 
nomic impacts of programs. To increase the clarity of this discussion, real 
examples of cost analysis from the literature are provided. 
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The Returns to Education 

This discussion is predicated on the belief that the returns to a college 
education far outweigh those of not attending a postsecondary institu- 
tion, and are measured in terms of the development of fiscal and social 
capital. From a purely financial standpoint, bachelor's degree recipients 
earn, on average, almost $20,000 per year more than individuals with 
only a high school diploma, with males earning about one third more 
than females (Census, 2004). Over the course of a lifetime, this translates 
into a net differential of approximately $770,000, before consideration of 
investment dividends. Professional students earn about 2.4 times three 
times that of high school graduates. 



Exhibit 1 . Personai Income for Full-Time, Fuii-year Workers Aged 25-64, by 
Educationai Attainment and Gender, 2003 



HI 

% 



Professional Degree 
Doctorate Degree 
Master's Degree 
Bachelor's Degree 

Associate's Degree 

Some College (no 
degree) 

HS Graduate 

Grade 9 to 12th Grade 
(no diploma) 

Less than 9th Grade 






eB/BL 








$10QOOO 









All Levels 



67,214 








$agi5E2 



5Q363 









41327 




■ 




$56616 1 




I Females 
I Males 



$0 $25,000 $50,000 $75,000 

Median Personal Income 



$ 100,000 



From a purely 
financial 
standpoint, 
bachelor's degree 
recipients earn, on 
average, almost 
$20,000 per year 
more than 
individuals with 
only a high school 
diploma. 



SOURCE: U.S. Census Bureau, Current Population Survey, 2004 Annual Social and Economic Supplement 
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Individuals who attend and graduate from college realize a number of 
short- and long-term benefits. The short-term consumption benefits of 
attending college include enjoyment of the learning experience, involve- 
ment in extracurricular activities, participation in social and cultural 
events, and enhancement of social status. Long-term or future benefits 
include higher lifetime earnings, more fulfilling work environment, better 
health, longer life, more informed purchases, and lower probability of 
unemployment (Bowen, 1980; Leslie & Brinkman, 1988; McPherson, 
1993). 

Over the course of the 1900s, more people enrolled and completed a post- 
secondary degree than ever before. Today, over 14 million students at- 
tend a postsecondary institution in the U.S., due in part to the under- 
standing that a college degree is the key to economic and social success in 
American society. This growth is not isolated. All groups, regardless of 
background, are going to college at higher rates than ever before. How- 
ever, access remains inequitable across these groups. Participation rates 
for students from low-income backgrounds and students of color signifi- 
cantly lag behind White and Asian students (Gladieux and Swail, 1999). 



All groups, 
regardless of 
background, are 
going to college at 
higher rates than 
ever before. 
However, access 
remains inequitable 
across these groups. 



Public Policy 

Public policy has long concerned itself with the plight of individuals from 
lower socio-economic levels. At the federal level, the 1940s GI bill and the 
1960's 'War on Poverty' are landmark examples of government safety net 
programs to ensure that individuals on the lower rungs of society are 
given some type of public support. The Higher Education Act of 1965 
(HEA) provided student aid and academic support programs targeted to 
low-income families and students of color. Today, the HEA provides over 
$40 billion to students in direct financial aid each year, plus several billion 
more in special programs for targeted groups. Additionally, state and lo- 
cal governments spend millions of dollars on special programs to facili- 
tate academic preparation and college access for underrepresented stu- 
dents. 

The federal government spends over $1 billion each year on two interven- 
tion initiatives to help students overcome social and cultural barriers to 
higher education. The largest appropriation ($833 million in EY2004) is 
through TRIO, which includes the pre-college programs Upward Bound 
and Talent Search. A second federal initiative and program, GEAR-UP 
(Gaining Early Awareness and Readiness for Undergraduate Programs), 
was created during the 1998 reauthorization of the Higher Education Act. 
GEAR-UP, which was appropriated for $298 million in PY2004, differs 
from TRIO by targeting a cohort of students in public schools for services 
and follow them through to graduation.^ 



1 See Swail and Pema (2002) for a more complete discussion of these and other programs. 
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Individual states have also created their own programs, such as Florida's 
CROP (College Reach-Out Program) and California's CalSOAP (Califor- 
nia Student Opportunity and Access Program). The private sector, 
through a variety of non-for-profit entities, is responsible for programs 
like the California-based MESA (Mathematics, Engineering, and Science 
Achievement) and the national "I Have a Dream Eoundation." And a few 
corporate foundations have created their own programs, such as "College 
Bound," sponsored by the GE Eund. 

Each year, programs like these compete for funds in both governmental 
and philanthropic arenas. Slicing the federal, state, or local budget pie is 
fraught with difficult choices of competing interests. Eor instance, only 1 
in 6 dollars of federal spending are considered discretionary in nature, 
meaning only 17 percent of the federal budget beyond the jurisdiction of 
the main departments of federal authority. Within this discretionary area, 
health services programs (e.g., NIH), research and evaluation (e.g., NSE), 
arts and humanities (e.g., NEA), and programming and research through 
the Department of Education each complete for a slice of the pie. Pro- 
grams must annually make their case for funding through the appropria- 
tions process, but in a shrinking economy, the stakes are greater. 

Although the federal government is a primary example, this story is no 
different at any other level of government. Programs must complete for 
funds in an environment where the competition can be fierce. Those who 
do well during an appropriations campaign often do so for two reasons: 
they effectively motivate their grassroots organizations and stakeholders 
to pressure the appropriations committee, and second, they effectively 
illustrate the importance and value of their program. Programs funded by 
philanthropic organizations also find themselves in a fix during tough 
times. The recent recession has forced many large philanthropic organiza- 
tions to reduce funding due to decreases in endowments. Thus, a poor 
economy invariably affects all programs, regardless of funding source. 
The policies of choice are difficult at the best of times — much more diffi- 
cult during economic downturns. 

Pre-college programs are not insulated from this reality. Half of all pre- 
college intervention programs receive federal funding, one quarter re- 
ceive state funding, and 20 percent receive funding from philanthropic 
organizations (Swail and Perna, 2000). Therefore, the issues discussed 
above do impact programs. 



Half of all pre- 
college intervention 
programs receive 
federal funding, one 
quarter receive state 
funding, and 20 
percent receive 
funding from 
philanthropic 
organizations. 
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Linking Program Costs with Effectiveness 

In an era where budgets are constricted and new sources of funding seem 
less apparent, the cost effectiveness of public programs is more important 
than ever. Interestingly enough, this seems to have had little impact on 
public policy. A number of studies have shown that several large-scale 
public programs, including Head Start, Upward Bound, and Success For All, 
have had marginal impact on students from an empirical standpoint. Still, 
each program has been rewarded with an increasing commitment of fed- 
eral funds because people 'understand' how these programs change lives, 
even if the research doesn't necessarily support this conclusion. This is 
hardly a tried-and-true model for the effective use of public funds. 

It must also be considered that the constriction of public funds will be- 
come more tenuous as time passes. The aging population will further 
constrain budgets at all levels of government, and is it plausible that pub- 
lic focus will shift to the elderly. So the competition for program funding 
is likely to intensify, not decrease, even given a recovered economy. This 
further underscores the need to look to this chapter's proposition regard- 
ing the impact of funding on program outcomes. 

Practice and theory must lead stakeholders to assume that increased pro- 
gram funding is associated with a greater chance of postsecondary access 
and success for participating students enlisted in these programs. Almost 
all rely on significant face-to-face contact with students, the most expen- 
sive type of intervention. Borrowing from Levine and Nidiffer (1996), at- 
risk and underrepresented students require "one arm around one child" 
in order to succeed. At the ConnectED pre-college program conference in 
2000, one of the success story participants, a first-generation college stu- 
dent from rural Virginia who graduated with honors from Harvard, com- 
mented that it took several pairs of arms to get her through. Thus, inter- 
ventions to support traditionally underrepresented students are costly 
and rely heavily on personnel. 



In an era where 
budgets are 
constricted and new 
sources of funding 
seem less apparent, 
the cost effectiveness 
of public programs is 
more important than 
ever. 



On a simplistic level, it is understood that a change in persormel funding 
or service delivery can have a net impact on student outcomes. Increased 
funding allows more of everything, while a reduction reduces the flow of 
services to students. Most practitioners will testify that staffing is critical 
to their efforts to serve students, and most acknowledge that their ability 
to impact students relies heavily on services provided through project 
staffing. However, cost efficiency in the analysis of program funding, 
staffing, and services on student outcomes must also be considered. In an 
economic sense, a dollar spent on staffing may not yield one dollar's 
worth of impact with regard to student outcomes. Similarly, that same 
dollar spent on web development, books and supplies, or transportation 
may not translate efficiently to the program's primary outcome. Thus, it is 
not guaranteed that program resources have a direct 1:1 ratio of spending 
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versus impacts. In some cases the ratio may be higher (more efficient) or 
sometimes lower (less efficient). 

Efficiency is a paramount concept for consideration. While educators are 
raised on the notion that program cuts are the antithesis of progress, the 
corporate world has operated on much different rules mandated by a 
competitive, global market. The 1990s were a decade of downsizing for 
corporations and businesses. The business/ industry sector found that 
strategic or surgical workforce cuts may generate a leaner, more efficient 
system capable of producing greater profits. It is possible that this phe- 
nomenon occurs in education as well, but there is little or no evidence 
substantiating this fact. Approximately 80 percent of public school budg- 
ets are spent on personnel. Intervention programs are no different, be- 
cause they also work in the business of direct services to students and 
families. Thinking in more pragmatic terms, funding for programs and 
strategies as discussed in this paper has a direct impact on the following: 

■ The number of students served; 

■ The number of teachers or instructors involved in a program; 

■ The hiring of administrative support, including a director; 

■ The coordination of volunteers; 

■ The rental of space for programming; 

■ The use of transportation (e.g., buses); 

■ Special programming (e.g., plays, etc.); 

■ Coaching materials for SAT / ACT test taking; and 

■ Counseling materials. 

Increases in funding allow for more of the above, and a reduction has the 
opposite effect. Because focus is rarely applied to cost analysis in educa- 
tion, the efficiency argument is also rarely considered. Discussion most 
often revolves around the impact of programs, usually through test 
scores or other concrete, tangible measures, but seldom is the extra step 
taken to look at long-term issues and program accountability. Perhaps 
stakeholders focus on services and process because they don't have the 
political longevity, inclination, or resources to focus on outcomes, cost, 
and effectiveness. Surely all intervention programs could be more effi- 
cient. There is no such thing as a truly efficient system, only that which is 
more or less efficient. 



Discussion most 
often revolves 
around the impact 
of programs, usually 
through test scores 
or other concrete, 
tangible measures, 
but seldom is the 
extra step taken to 
look at long-term 
issues and program 
accountability. 



Moving from Belief to Empiricism 

To move from the theoretical to the practical world, tangible, empirically- 
based models to measure program impact and efficiency must be devel- 
oped. Without models, policymakers will be forced to make prudent pol- 
icy decisions based on limited information. As well, it can't be assumed 
that more money allocated to education will have a significant impact on 
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student outcomes unless their exists evidence of the linkage between the 
two (Hummel-Rossi and Ashdown, 2002, p. 1). 

Three questions must be answered to produce the information desired. 
Does the program produce the desired impact that it was designed to 
produce? Does the benefit of the program outweigh program costs? And 
third, does the program offer the most cost-effective and appropriate way 
of reaching the desired goals? 

Unfortunately, the ability to answer these questions within the scope of 
pre-college intervention programs is severely hampered by the lack of 
data and unsophisticated research methodologies (Gandara, 2002; Gan- 
dara and Bial, 1999; Swail and Perna, 2002; Tierney, 2002). According to 
Gandara, "college interventions suffer from a serious lack of rigorous 
evaluation, in spite of the millions of dollars that are invested in them an- 
nually" (2002, p. 97). 

The first question is the foundational question for all analysis. One simply 
must know how the program impacted students. But that cannot readily 
be answered because (a) resources are rarely applied to evaluation and 
analysis, and (b) it isn't often deem as a priority. When traveling down 
the evaluation road, researchers and practitioners typically default to the 
more simplistic measures of student outcomes, such as test scores, atten- 
dance patterns, and retention in school. By doing so, they step away from 
the more difficult work of addressing other potential benefits of interven- 
tion programs, including medium and long-term effects that are often less 
tangible. Many programs designed to help students get into college do 
not collect accurate data on student outcomes (Gandara and Bial, 1999; 
Swail and Perna, 2002), and information about program costs is often not 
reported or collected from programs, even in large-scale educational 
evaluations (Karoly et al., 2001). 

Cost analysis is rarely used in education, despite significant expenditures 
within an environment that is significantly more interested in program 
outcomes and effective use of taxpayer dollars than any previous genera- 
tion (Hummel-Rossi and Ashdown, 2002). As well, understanding the 
role of cost is an important factor in program design, population targeting 
strategies, and implementation (Karoly et al., 2001). Questions of costs 
and effectiveness are asked of all social programs at one time or another, 
and society is only beginning to think more of this line of query in educa- 
tion. 



Does the program 
produce the desired 
impact that it was 
designed to produce? 



Does the benefit of 
the program 
outweigh program 
costs? 



Does the program 
offer the most cost- 
effective and 
appropriate way of 
reaching the desired 
goals? 



Cost Analysis Defined 

Cost analyses provide a mechanism to compare program costs to the 
benefits due to the program, and can be used to promote fiscal account- 
ability, set priorities when resources are limited, and act as an effective 
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tool to persuade legislators and potential funders of the importance of the 
program (Sewell and Marczak, 2002). Additionally, cost analysis can help 
policymakers and program directors estimate a program's costs and 
benefits before implementation, improve the understanding of program 
operation and underscore the most cost-effective components, and iden- 
tify unexpected costs. Disadvantages include the high level of technical 
skill required, disagreements over the benefit of these types of analyses, 
and the difficulty in assigning monetary values to qualitative goals. Im- 
portant to note is that, in the end, cost analyses cannot suggest whether a 
program or strategy has exhibited a desired impact. It can only describe 
the costs associated with that impact (Rice, 1996). 

Cost-benefit and cost-effectiveness analyses are forms of efficiency 
analysis that attempt to define the benefit of a program or policy versus 
the cost (Rossi and Freeman, 1993). Although the two forms of analyses 
are different in scope, the lines between the two often get blurred in the 
literature. A cost-benefit analysis evaluates the costs and benefits of a pro- 
gram in dollars and then compares the two, and is essentially interesting 
in answering the question, "is this program worth doing?" 

Three analytical steps make up a cost-benefit analysis, although there are 
many considerations that must be made within each of these steps. The 
first step is to evaluate the costs of running the program. Second is the 
evaluation of program benefit. And third is the determination of whether 
the calculated benefit outweighs the accounted program cost. This is usu- 
ally done in the form of a cost-benefit ratio, which is nothing more than 
product of dividing the benefit by the cost. For example, if program cost 
is accounted at $10,000 and the determined benefit is $40,000, then: 

$4Q0Q0 

CBRatio = =40 

$]Q0Q0 



Cost-benefit and 
cost-effectiveness 
analyses are forms 
of efficiency 
analysis that 
attempt to define 
the benefit of a 
program or policy 
versus the cost. 



In the example above, the cost-benefit ratio is 4.0, meaning that the pro- 
gram benefits outweigh program costs by a 4-to-l ratio. Put another way, 
the returns to the program are four times that of the investment. 

Cost-benefit analysis is generally used to determine the cost versus bene- 
fit of a single program, and may be used to determine either the fiscal 
benefit to the program or institution (e.g., cost savings) or the fis- 
cal/ societal benefit to the individual and society (e.g., life-time earnings). 
With respect to pre-college outreach programs, a cost-benefit analysis can 
be utilized to determine whether the individual and societal benefits of a 
community youth mentoring program outweigh the costs of program de- 
livery. 
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A cost-effectiveness analysis differs from a cost-benefit analysis by con- 
sidering the relative costs and benefits of a number of alternative pro- 
grams (Rice, 1993). With cost-effective analysis, researchers are generally 
interested in answering the question, "which of these interventions is 
more efficient in terms of its use of resources?" Benefits are assumed in a 
cost-effectiveness analysis, such as when the cost-effectiveness of Pro- 
gram A versus Program B are compared. It is assumed that both pro- 
grams have about the same impact, but stakeholder interested is in know- 
ing which program does so in a more efficient manner. 

Cost analysis is, on a simplistic level, an example of proper use of pro- 
gram budgeting and account practices (Kettner, Moroney, & Martin, 
1990). Instead of credits and debits, the work focuses on costs and bene- 
fits. The cost side of the equation involves the allocation of dollar values 
to all program inputs and resources, such as staffing, logistics, and mate- 
rials. The benefit side is infinitely more complex due to the difficulty in 
allocating resources to future situations. 

Depending on the scope of the study, calculating costs and benefits can be 
either simple or complex. The further out the benefit to be calculated, the 
more complex it becomes because social capital must be calculated. For 
instance, to calculate the costs associated with a pre-college program, one 
would sum up tangible program costs such as staffing, overhead charges 
associated with program space, materials, transportation, and perhaps 
additional costs for special events, such as field trips. One could also add 
the opportunity costs associated with participating in the program, such 
as immediate earnings that could be earned during program hours. 

On the benefit side, one must then sum up the tangible and less tangible 
benefits, immediate and future oriented. These may include increased 
earnings for students who persist and earn a bachelor's degree, increased 
tax revenue, and the reduction in incarceration rates and recidivism. 
Exhibit 2 illustrates the items used in a 1971 cost-benefit analysis of Up- 
ward Bound participants. 



With cost-effective 
analysis, researchers 
are generally 
interested in 
answering the 
question, "which of 
these interventions 
is more efficient in 
terms of its use of 
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Exhibit 2. Description of costs and benefits associated with Upward Bound 
cost-benefit analysis (1971, Garn). 

QeIs Benefits 



• Additional tuition costs required of Upward • 

Bound students due to higher rates of post- 
secondary attendance. 

• Additionai expenses of Upward Bound stu- 
dents whiie in coliege. * 

• Earnings foregone by Upward Bound stu- 
dents whiie in coliege. 

• T ransfer income over the iifetime foregone 
by Upward Bound students (e.g., unempioy- 
mentand weifare). 

SOURCE: (Rossi and Freeman, 1993, p. 385) 

Variables such as foregone earnings, unemployment and welfare costs, 
and after-tax lifetime income involve complex calculations. Because the 
identification of costing of benefits can be subjective to a certain degree, 
the outcomes of a cost-benefit analysis are often debated. 

Cost analysis takes many forms and is peculiar to the type of program 
being assessed. Again, many of these analyses are very complex. The 
scope of this chapter does not allow for an indepth investigation into spe- 
cific cost analysis treatments. However, to provide a better understanding 
of how cost analysis works in the real world, it is worth reviewing a few 
examples from the research literature. 



Increased after-tax lifetime incomes. 

Stipends paid to participants during the 
program. 

Scholarships and other grants received 
by Upward Bound students in college. 



Case Study I. The Supplemental Instruction Experience 

The Supplemental Instruction (SI) program, developed at University of 
Missouri-Kansas City (UMKC) in 1974, provides academic support to 
students through specific course work. While not labeled as a tutoring 
program, SI contributes tutoring-like experiences for students on campus. 
The UMKC SI program, currently in use at over 1,100 institutions na- 
tionwide, provides three weekly sessions of academic support for stu- 
dents begirming the first week of class and is open to all students on a 
voluntary basis. 

Participation in SI at UMKC has been equal across all levels of students, 
with the same number of students from low and high ACT composite 
score quartiles enrolling. Still, the program targets classes with 30 percent 
or higher rates of D and F grades. By developing the students' concepts of 
how to learn with knowing what to learn, the SI program has been 
known to reduce the number of Ds and Fs in addition to the number of 
course withdrawals by up to 50 percent. 
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Following the 1995-96 academic year, SI conducted an internal cost- 
benefit analysis of their program (UMKC, 1997). Of the 3,655 students en- 
rolled in 41 selected courses at UMKC in 1995, 40 percent (1,454) of the 
students participated in SI at an average cost per student of $46.89. Ac- 
cording to their analysis, SI students re-enrolled in school and graduated 
at rates 10 points higher than students who couldn't take SI due to sched- 
uling conflicts. Based on this rate, SI infers that 145 students would have 
dropped out if not for SI (1,454 students x 10% = 145). Given that the av- 
erage undergraduate student spends $1,750 on tuition, fees, and other 
expenses each semester, those 145 students provided a revenue increase 
of $253,750 to the university (145 x $1,750 = $253,750). This analysis only 
accounts for one cohort of students. Taking into consideration the full 
impact of four armual cohorts of freshman students (5,302 students in to- 
tal), the net retention impact at any one time is argued to be 530 (using 
the 10 percent retention rate) additional students due to SI. The financial 
impact resulting from SI is almost $1 million each year ($1,750 x 530 = 
$927,500). What is not accounted for here are the costs associated with 
recruitment and admissions services that must be applied to each admit- 
ted student. 



Case Study II. The Perry Preschool Program 

The cost-benefit analysis of the Perry Preschool program is a benchmark 
case in the research literature (Karoly et al., 2001; Borman & Hewes, 2001; 
Schweinhart et al., 1993). Based in Ypsilanti, Michigan, the program 
served 58 African American children from low socio-economic status 
backgrounds and with low IQ scores between 1962 and 1967. Program 
participants began at age three and received two years of services while 
four-year olds received one year. The Perry Preschool program provided 
high quality staffing and learning opportunities, with low pupil-teacher 
ratios. Although the program was small, a good portion of the interest 
generated by this study stems from the fact that participants from the ex- 
perimental and control groups were followed through age 27. 

Impact of the program was measured through several variables. Program 
participants realized IQ scores 12 points higher than control participants, 
and academic achievement was also significantly enhanced through pro- 
gram participation. Although there were no differences in postsecondary 
participation, the final follow-up (age 27) found lasting differences in 
employment, welfare, and crime. 

In his original cost-benefit analysis of the program, Barnett (1993) found 
that benefits to society exceeded program costs by a factor of seven-to- 
one. Karoly et. al (1998), in a re-analysis of the program data, found that 
the program reduced use of special education, reduced grade retention, 
increased taxes due to higher employment, lessened reliance on welfare 
programs and funding, and reduced justice system costs (Karoly et al.. 



Educational Policy Institute 



12 



Value Added 



2001). Exhibit 3 shows Karoly et al.'s cost-benefit comparison of a pro- 
gram cost per child of $12,148 to a total benefit of $49,972, for a net benefit 
of $37,824. While not reaching Barnett's claim of a seven-to-one benefit, 
Karoly et al.'s calculation still manages a 4 to 1 cost-benefit ratio. 



Exhibit 3. Costs and Benefits: The Perry Preschool Program 





Doiiars perChiid 




Due to Mother 


Due to Chiid 


Total 


Program cost 






12,148 


Savings to government 






25,437 


Reduction in education 
services 


* 


6,365 




Reduction in heaith services 


* 


* 




Taxes from increased 
empioyment 


* 


6,566 




Reduction in weifare cost 


* 


2,310 




Reduction in criminai justice 
cost 


* 


10,195 




Additionai monetary benefits 






24,535 


Increase in participant income 
net of weifare ioss 


* 


13,846 




Reduction in tangibie iosses to 
crime victims 




10,690 




Totai benefits 






49,972 


Net benefits 






37,824 


SOURCE: Karoly, Lynn A., Kilburn, M. Rebecca, Bigelow, J a mes H., Caulkins, J onathan P., Cannon, J ill S., & Chiesa, 
lames (2001). Assessing Costs and Benefits of Early Childhood Interventions. Santa Monica, CA: RAND, p. 57. 


NOTE: ♦ = not measured. All amounts are in 1996 dollars and are the NPV of amounts overtime where future values are 
discounted to the birth of the participating child, using a 4 percent annual real discount rate. 



Case Study III. Success For All 

Success for All (SEA) is an early childhood reading program implemented 
in more than 1,500 schools throughout the United States (Matthews, 2002, 
p.33). A recent cost-effectiveness analysis of the program (Borman and 
Hewes, 2001) focused on answering two research questions: what is the 
relative impact of the program on student success, and what is the cost of 
program delivery? 

With regard to the former, Borman and Hewes tracked the educational 
outcomes of entering students (grades 1 through 5) through the eighth 
grade, using program data plus data from two norm-referenced standard- 
ized tests: the California Achievement Test (CAT) and the Comprehen- 
sive Test of Basic Skills (CTBS). Using a quasi-experimental approach, the 
researchers were able to discern the academic impact of SEA to students 
in a control group (thus making it a cost-effectiveness analysis). 



Educational Policy Institute 



13 




Value Added 



For program delivery, the researchers collected several cost estimates. 
Basing their analysis on Levin & McEwan's Ingredients Model (2001), the 
analysis summarized the total and marginal costs for program delivery. 
Marginal costs were limited to training, materials, and professional de- 
velopment. Costs associated with personnel costs within the Baltimore 
school system were also estimated. 

The researchers acknowledged that SEA is an expensive reform model to 
implement. However, the purpose of the analysis was to determine 
whether SEA was any more expensive than the alternatives of having 
students repeat grades or enrolling in special education. The researchers 
concluded that the mean cost of reaching the eighth grade for SEA stu- 
dents was $70,428, compared to $68,850 for control students. This differ- 
ence is not statistically different. 

Based on further adjustments and calculations, the authors found that 
SEA had a larger impact based on the cost of the program than published 
studies of other programs. However, a recent re-analysis found that the 
average SEA student failed to reach grade-level performance by the end 
of the third grade, and students continued to fall further behind national 
norms through SEA. "By the end of the fifth grade, they were almost 2.4 
years behind" (Matthews, 2002, p. 35). Given that the original Borman 
and Hewes analysis found the program to be cost effective AND have 
desirable results, these new findings suggest a different conclusion, and 
also illustrate how findings from alternative cost analyses can be debated 
and contradictory. 



Exhibit 4. Per-Pupil Expenditures and Sustained Effect Sizes for Four Educa- 
tional Interventions. 





Annual PPE 
(In 2000 
dollars 


Years of Inter- 
vention 


Total PPE (In 
2000 dollars) 


Sustained 
Effect Size 


Effect per $1,000 


Success for All 


$603 


4.56 


$2,749.68 


0.27 


0.1 


Class Size Reduction to 15 












Nye etal. 


$998 


4 


$3,992.00 


0.32 


0.08 


Finn etal. 


$998 


4 


$3,992.00 


0.22 


0.06 


Perry Preschool 


$8,929 


2 


$17,858.00 


0.51 


0.03 


Abecedarian, Preschool Only 


$10,496 


4.5 


$47,232.00 


0.53 


0.01 


Source: Borman and Hewes (200X), p. 34. 
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Cost Analysis and Public Policy 

Focusing exclusively on cost can also have negative ramifications. Ac- 
cording to Rice (1996), while cost analysis can help determine feasibility 
within a limited resource pool, it provides no information on the effec- 
tiveness of the program in meeting the desired needs. "To the degree that 
an intervention is ineffective, adopting it exclusively on the basis of its 
relative cost may be a waste of valuable resources, potentially translating 
into exceedingly high long-term costs" (Rice, 1996, p. 34). 

Policymakers often look at per pupil expenditures as a simple default for 
cost analysis. As understood, there are great disparities in per pupil ex- 
penditures (PPE) across counties, states, and regions. For example, the 
median average PPE for U.S. public schools in 2002 was $6,657 (NEA, 
2002). North Dakota, on one other hand, spent $4,426 per student, com- 
pared to a whopping $12,345 in the District of Columbia. Valuing Rice's 
(1996) statement of the break between feasibility and effectiveness, it is 
clear that, although the District of Columbia spends almost three times 
the PPE than North Dakota, per-pupil expenditures do nothing to explain 
why the latter scores significantly higher on NAEP tests, on average, than 
DC students (NAEP, 2001). Nor can it be assumed that DC spends three 
times North Dakota on student learning. Therefore, care must be taken 
when using a single number, such as a PPE, in a costing model without 
taking other factors into consideration. 

Much damage can be done when cost analysis data are misused. Eor in- 
stance, to suggest that a program is inefficient or too costly to maintain 
based solely on the number of individuals who meet the final program 
criteria, without consideration of those who benefit from the program but 
do not reach the final stated goals, is a careless use of data. One must al- 
ways attempt to build in to the analysis all unintended positive im- 
pacts — or positive externalities — of program. A simple example is the 
analysis of postsecondary outcomes. Consider the example of a partici- 
pant in a college intervention program who fails to complete a bachelor's 
degree but does manage to complete an associate's degree. It is quite 
likely that this individual will lead a productive, tax-paying, community- 
serving life. Eailure to include this individual as a positive benefit of the 
program in a cost-benefit analysis negatively skews the cost-benefit ratio. 

One must also be mindful of misinterpreting a positive cost-benefit analy- 
sis as an implication of program feasibility or effectiveness, or the oppo- 
site, that a positive impact suggests that the program is cost effective. 
These misconceptions are common in the literature, where research arti- 
cles point to the "cost effectiveness" of a program, with no evidence of 
evaluating program inputs as they relate to outputs. Just because a pro- 
gram has a positive impact does not infer that it is cost effective or effi- 
cient. It only implies a positive impact. 



Much damage can be 
done when cost 
analysis data are 
misused. 



One must also be 
mindful of 
misinterpreting a 
positive cost-benefit 
analysis as an 
implication of 
program feasibi lity 
or effectiveness. 
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And finally, cost and impact analyses may be largely ignored in the pub- 
lic policy arena (Anderson, 1993). When an evaluation has a negative 
finding on a program, it is common for organizations and individuals to 
critique the evaluation on methodological grounds and attempt to negate 
the evaluation. This was illustrated recently when a 1999 study of Up- 
ward Bound was released with findings that didn't paint an overall posi- 
tive picture of the large-scale federal program (Myers and Schirm, 1999). 
Critics quickly pointed to methodological problems associated with the 
choice of control group participants, putting the evaluation into question. 
Another federal program. Head Start, has traditionally battled poor re- 
search findings. As with Upward Bound, critics quickly lambasted these 
evaluations. Head Start remains a viable and important program for pre- 
school youth that enjoys bipartisan support in Congress. 

Thus, even thoroughly and appropriately conducted evaluations and as- 
sessments rarely determine the final outcome for a program. Anderson 
(1993) suggests that he is unable to think of a governmental program that 
was terminated "solely as a consequence of an unfavorable systematic 
evaluation" (Anderson, 1993, p. 292). 



Suggestions for Improving the Quality of Inquiry 

For those who are deeply interested in cost analysis as a tool to better un- 
derstand their program and to provide evidence of their program's social 
impact, strongly recommended readying includes Barnett's (1993) nine- 
step process for cost-benefit analysis and Rice's (1997) template-driven 
model for unpacking costs and weighting benefits. It is worth reminding 
readers that these can become very complex, and outside expertise may 
be required. 

Whether one actively pursues a cost analysis, or whether one is reviewing 
information from a cost analysis, four issues are worthy of consideration: 

Quality of information. An analysis is only as good as the data. For the 
cost side, taking numbers directly from budget information may not be 
accurate. Researchers may have to determine actual costs associated with 
program operation from a variety of sources. On the benefit side, the fur- 
ther out one gets from the program (in terms of time), the more difficult it 
is to predict data that accurately depicts program benefits. The benefit 
portion of the analysis can require subjective decisions about the weight- 
ing or valuing of data, thus must be considered carefully. 

Short vs. long-term impact. Do stakeholders know what happens to stu- 
dents well after the intervention is complete? Given that a number of 
studies find that the effect of interventions often fade over time (Borman 
and Hewes, 2001; Currie and Thomas, 2000), it is important to understand 
this phenomena, as it could impact the design and delivery of program 
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services. Unfortunately, long-term analysis often requires longitudinal 
analysis of participants, which is tremendously costly. Additionally, de- 
fining the scope of the study at the outset will determine whether this is a 
short-term look or a long-term perspective. 

Tangible vs. intangibles. Counting who completes, who graduates, or 
who scores well is generally not difficult work in itself. Calculating the 
impacts or defining the costs on seemingly intangible items, such as the 
"cost" of volunteers, or the societal benefit of an intervention, is much 
more complex. "Although it is often difficult to assign a value to many of 
the inputs and outputs of educational interventions, their inclusion in the 
total cost and effectiveness estimates is essential. This gives policymakers 
a realistic sense of the overall cost of the intervention, not just to the 
budget but to the community" (Rice, 1996, p. 37). 

Micro vs. Macroeconomics. Researchers are generally inward rather than 
outward looking. Sometimes limited research budgets are responsible, 
but other, external factors in the analysis must be considered. Otherwise, 
the research is encapsulated by the researcher's protected notion of 'what 
is,' with little regard for the impact of the program on other parts of soci- 
ety. Thus, the researcher must first define whether the interest is in de- 
termining the immediate, internal program effectiveness and benefits, or 
whether the long-term impact on the individual and society must be 
brought into the analysis. 
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Final Comments 

With accountability the catch-phrase of the day, knowledge of program 
effectiveness and impact is becoming more critical, especially in light of 
recent budget crunches at all levels of government. The need to produce 
evidence of program impact is important for sustained fiscal support. 

Pre-college programs often provide meaningful, individualized contact 
between a youth and an adult that is not typically available in the com- 
munities or schools from which they hail. For public policy, this is the 
most difficult type of policy to craft. As Levine and Nidiffer (1996) state, 
getting poor people into college "is retail, not wholesale, work in the 
sense that it requires intensive involvement with individuals." This type 
of effort requires personnel-driven budgets, which, as discussed, are typi- 
cally the most expensive line item in the program budget. Therefore, in- 
creases in funding almost assuredly impacts the level or scope of service 
provided. 

In the final analysis, costs matter. Interventions that expend $4,500 per 
student undoubtedly offer more robust services than those that average 
$300. All programs could stand a little efficiency testing, and perhaps 
there are more efficient ways to help needy students prepare for and ac- 
cess college than now exist. For now, stakeholders must redouble their 
effort to uncover the findings from current programs and promote con- 
tinuous improvement of these efforts. For this to be realized, governmen- 
tal agencies and policymakers must demand more information about 
how these programs operate and back it up with the necessary funding to 
make that happen. Unfunded mandates won't cut it. Philanthropic or- 
ganizations, too, must demand a higher level of empirical evidence and 
support the needs of programs to that end, and also provide the financial 
support to make it happen. And program practitioners must push beyond 
current knowledge and practice and begin thinking and practicing high- 
level analysis and program management well before it comes to them in 
stated policy. 

Of course, cautious must continue to be exercised about the use of num- 
bers and statistics in defining social interventions. Einstein once said that 
not everything that can be counted counts, and not everything that counts 
can be counted. The use of cost analysis to explore the effectiveness of 
programs is an important tool, but results used out of context or in a 
truncated fashion can be more dangerous to the development of prudent 
public policy. 
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